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Introduction 


What  is  the  proper  role  of  men  and  machines  in  decision  making 
systems?  Two  competing  philosophies  have  taken  extreme  positions. 

The  first  is  that  the  role  of  man  in  intellectual  systems  is  to  do 
intellectual  work.  In  particular,  the  role  of  man  in  a  decision  making 
system  is  to  make  decisions.  However,  the  fact  that  man  is  in  many 
situations  a  suboptimal  information  processor  and  decision  maker,  to¬ 
gether  with  the  power  of  the  computer  hardware,  has  led  some  to  pro¬ 
pose  the  opposite  viewpoint,  i.e.,  that  man  has  no  role  at  all.  Thus, 
operations  researchers  have  developed  fully  automatic  systems  to  perform 
decision  tasks- -systems  that  have  completely  bypassed  man.  So  we  have 
an  antinomy  between  fully  automated  systems  on  the  one  hand  and,  on 
the  other,  systems  in  which  men  perform  the  intellectual  tasks  while 
the  computer  keeps  records,  manipulates  displays,  and  so  on. 

Both  extreme  formulations  are  absurd.  Instead  of  arguing  that  men 
are  superior  to  machines  or  that  machines  are  superior  to  men  we  should 
be  interested  in  the  right  combination  of  men  and  machines  for  a  par¬ 
ticular  intellectual  task.  We  have  to  find  out  what  subtasks  within 
each  intellectual  j'>b  men  perform  better,  what  subtasks  machines  perform 
better,  and  how  to  blend  men  and  machines  for  each  job. 

Psychologists  in  che  area  of  decision  theory  have  developed  procedures 
whereby  men  break  complex  problems  into  simple  component  parts,  then  use 
expert  judgment  to  evaluate  those  components,  and  then  the  machine  uses 
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rules  from  decision  theory  to  combine  the  component  assessments  into  a 
global  conclusion. 

I 

One  natural  decomposition  is  into  the  tasks  of  diagnosis  and  decision  i 

making.  Diagnosis  is  finding  out  what  the  situation  is  and  decision  making  j 

i 

is  choosing  what  to  do  about  the  situation.  For  diagnosis,  men  judge  the  \ 

\ 

impact  of  each  piece  of  evidence  and  machines  put  the  implications  of  j 

those  pieces  of  evidence  together.  For  decision  making,  men  evaluate  the  I 

utility  of  the  possible  outcomes  of  each  course  of  action,  whereas  machines  j 

make  the  actual  decisions.  That  is,  they  decide  which  course  of  action  to  I 

i 

1 

i 

take.  These  particular  functions  are  assigned  to  machines  because  we  know  j 

of  good  algorithms  for  them.  Bayes's  theorem  is  the  appropriate  algorithm  \ 

for  diagnosis  and  the  principle  of  maximizing  expected  utility  is  the  ; 

i 

algorithm  for  making  decisions.  Both  usually  require  only  trivial  computations.  j 
But  these  computations  depend  on  the  proper  inputs,  and  that  is  what  men 
provide.  j 

Man's  primary  inputs  to  the  machine  are  estimates  of  utilities  (numbers  j 

3 

reflecting  worth  or  attractiveness)  and  probabilities  (numbers  reflecting 

] 

restive  '.ikelihood  of  occurence).  These  two  kinds  of  numbers  are  a  ! 

i 

; 

reflexion  of  the  principle  that  every  decision  should  depend  upon  the  ! 

j 

answers  to  two  questions — what  is  at  stake  and  what  are  the  odds.  The  j 

problem  is  to  find  ways  of  eliciting  suitable  judgments  of  these  numbers.  That 

.  i 

was  the  purpose  of  research  conducted  under  this  contract.  The  first  section  ] 
below  describes  research  on  probability  assessment,  the  second  describes 
research  on  utility  assessment,  the  third  describes  on-line  attempts  to  try 
out  the  results  of  the  research  on  actual  naval  problems,  and  the  final  ! 
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section  describes  the  initial  development  of  a  handbook  designed  to 
aid  users  of  decision  analyses. 

Assessment  of  Probability 

A  Scoring  Rule  to  Train  Probability  Assessors  (037230-4-T) . --  Two 
experiments  showed  that  a  test  based  on  a  proper  scoring  rule  could  im¬ 
prove  probability  assessments.  The  test  contained  three  se-sions  of  factual 
questions.  The  Ss  indicated  which  of  two  answers  was  more  likely  to  be 
correct  for  each  question  and  then  used  odds  assessments  as  an  indication 
of  how  sure  they  were  about  each  answer.  The  first  and  third  sessions  used 
the  same  set  of  50  questions;  Ss  were  not  informed  whether  their  answers 
were  correct.  The  second  session  used  a  different  set  of  75  questions;  Ss 
were  informed  whether  each  answer  was  correct  and,  in  addition,  received  a 
score  based  upon  a  proper  scoring  rule  (a  function  that  increases  the  score 
earned  when  higher  Js  are  assigned  to  the  correct  answer) .  One  experiment 
used  college  students  and  the  other  used  intelligence  analysts.  In  both 
cases  most  Ss'  earned  higher  scores  in  the  third  than  in  the  first  session, 
presumably  as  a  result  of  experience  with  the  scoring  rule  during  the  second 
session.  This  research  is  described  in  detail  in  the  technical  report. 

Experiments  were  also  conducted  on  response  modes  and  probability  diagrams 
but  has  not  as  yet  yielded  results  that  are  sufficiently  meaningf\I  to  write 
up  in  a  technical  report.  Consequently,  it  is  anticipated  that  research  on 
these  topics  will  continue  next  year.  In  addition,  these  two  topics  are 
included  as  chapters  in  the  handbook  described  below. 
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Assessment  of  Utility 

Multi-dimensional  Value  Assessment  for  Decision  Making  (037230-2-71.-- 
Dccision  analysis  is  a  tool  that  can  be  used  to  improve  the  quality  of 
complex  decisions  in  an  uncertain  environment.  A  decision  analysis  is 
constructed  by  specifying  alternative  courses  of  action  and  the  possible 
consequences  of  action,  Each  of  the  consequences  is  evaluated  in  terms  of 
its  relative  probability  of  occurrence  and  its  value  to  the  decision 
maker  if  it  should  occur. 

Decision  analysis  has  been  used  primarily  in  business  settings  where 
values  of  consequences  can  be  measured  in  terms  of  dollars.  In  non-business 
environments,  however,  non-monetary  criteria  may  be  of  paramount  importance. 
The  situation  is  further  complicated  if  relevant  values  vary  along  more 
than  a  single  dimension.  This  technical  report  reviews  the  psychological 
literature  on  the  problem  of  assigning  numerical  values  when  several  value 
attributes  (or  critoria)  are  relevant  to  the  decision  maker. 

This  literature  is  reviewed  from  both  a  descriptive  and  a  normative 
point  of  view.  That  is,  how  do  people  evaluate  multi-attribute  objects, 
and  how  should  they?  A  simple  weighted  average  provides  a  good  description 
of  how  people  do,  in  fact,  make  such  evaluations.  The  weighted  average 
approach  is  also  appropriate  for  many  normative  purposes  and  several  pro¬ 
cedures  for  making  this  evaluation  process  explicit  arc  discussed  and 
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criticized. 


The  Application  of  Multi-attribute  Scaling  Procedures  to  the 


Development  of  Indices  of  Value  (037230-1-T) . —  Multi-attribute  scaling 
procedures  were  applied  to  a  non- laboratory  problem— to  the  measurement 
of  water  quality.  Several  different  variables  (such  as  nitrate,  fecal 
colof^rms,  turbidity  and  dissolved  solids)  contribute  to  water  pollution. 

It  is  possible  to  obtain  physical  measures  of  each  of  the  variables,  but 
no  physical  model  exists  for  combining  the  measures  into  an  overall  index 
of  quality. 

The  multi-attribute  scaling  procedures  were  applied  to  this  task  by 
assessing,  from  water  quality  engineers,  (1)  judgments  about  which  variables 
should  be  included  in  the  index,  (2)  the  type  of  rule  for  combining  the 
variables,  (3)  the  relative  importance  weights  of  the  different  variables, 
and  (4)  a  curve  describing  the  functional  relation  between  water  quality 
and  each  variable. 

Water  quality  indices  were  obtained  for  two  specific  purposes- -for 
"public  water  supply"  and  for  "fish  and  wild-life".  The  experimenter  used 
a  modified  Delphi  procedure  for  obtaining  concensus  among  the  engineers  for 
each  of  the  indices.  Even  after  the  applications  of  the  Delphi  procedure, 
the  engineers  disagreed  on  the  importance  weights,  so  a  sensitivity  analysis 
applied  the  different  indices  to  actual  measurements  on  samples  of  river 
water.  This  analysis  indicated  that  the  disagreement  about  the  weights 
was  not  crucial  to  the  measurement  of  water  quality.  In  fact,  a  major 
conclusion  of  this  research  is  that  the  multi -attribute  scaling  procedures 
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are  sufficiently  robust  so  that,  while  great  care  should  be  used  in 
determining  the  purpose  for  which  the  index  will  be  used  and  in  selecting 
variables  for  inclusion,  relatively  little  time  and  effort  need  be  invested 
in  resolving  small  differences  among  quality  functions  and  among  relative 
weights.  The  technical  report  describes  this  research  in  detail;  it  is 
Michael  F.  O'Connor's  PhD  dissertation. 

Ratio  Versus  Magnitude  Estimates  of  Importance  Factors  (037230-3-T) 
Optimal  decision  making  requires  that  the  decision  maker  trade  off  various 
goals  or  objectives  against  one  another  in  selecting  a  course  of  action. 

This  experiment  compared  two  procedures  for  assigning  importance  weights 
to  objectives.  The  first  used  magnitude  estimates  and  the  second  ratio 
comparisons  of  importance.  The  ratio  procedure  produced  substantially 
greater  discrimination  between  the  importance  weights  assigned  to  objectives 
than  did  the  magnitude  estimation  procedure. 

A  sensitivity  analysis  revealed,  however,  that  additive  evaluation 
models  were  relatively  insensitive  to  the  differences  between  the  importance 
weights  produced  by  the  two  procedures.  Additive  models  based  upon  the 
two  types  of  weights  assigned  very  similar  overall  values  to  alternatives. 
The  technical  report  describes  the  research  in  detail. 

Four  Methods  for  Assessing  Multi -attribute  Utilities:  An  Experimental 
Validation  (037230-6-T) . —  In  choosing  between  altervatives  characterized 
by  multiple  value  relevant  attributes,  decision  makers  must  typically 
trade  off  one  attribute  against  another.  Previous  research  has  shown  that 
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as  the  number  of  attributes  describing  alternatives  becomes  large,  this 
subjective  trade-off  process  becomes  increasingly  subject  to  error  and 
that  decision  makers  tend  to  ignore  value-relevant  considerations.  These 
shortcomings  of  the  subjective  evaluation  process  have  been  related  to 
more  general  limitations  on  the  human  capacity  to  process  information. 
Decomposed  evaluation  procedures  seek  to  improve  upon  subjective  evaluation 
by  dividing  the  overall  evaluation  problem  into  a  set  of  simpler  subtasks, 
each  of  which  is  well  within  the  judgmental  capacities  of  the  decision  maker. 

The  first  section  of  this  technical  report  discusses  the  theoretical 
basis  for  multi-attribute  value  assessment  and  concludes  that  while  additive 
evaluation  models  should  be  appropriate  for  most  riskless  decisions,  non¬ 
additive  models  will  frequently  be  required  for  decision  making  under 
uncertainty.  The  second  section  discusses  the  sensitivity  of  evalution 
models  to  assessment  errors.  The  third  section  describes  four  procedures 
for  constructing  a  decomposed  evaluation  model.  The  fourth  section  treats 
the  general  problem  of  validating  evaluation  procedures.  And  the  final  two 
sections  discuss  two  experiments  which  demonstrated  that  all  four  of  the 
decomposition  procedures  described  can  provide  an  appropriate  measure  of 
subjective  valu->.  This  technical  report  is  Gregory  W.  Fischer's  PhD 
dissertation. 

On-Line  Research 

Considerable  time  was  spent  with  naval  analysts  in  OP  942U  during  the 
period  of  this  contract  in  an  attempt  to  apply  various  procedures  of 
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probability  assessment  to  the  problem  of  submarine  surveillance.  One 
of  the  more  important  results  of  this  effort  was  the  development  of  a 
procedure  for  blending  historical  frequency  information  with  expert  judg¬ 
ment  in  a  Bayesian  analysis.  This  procedure  has  now  been  incorporated 
in  the  Bayesian  chapter  of  the  handbook  described  below.  The  following 
exerpt  from  the  30  September  1971  annual  report  illustrates  how  beta 
distributions  are  used  with  this  procedure. 

The  following  actual  case  study  is  described  in  hypothetical  terms: 

"a  submarine  has  been  sighted  leaving  the  Mediterranean  Sea.  The  analysts 
were  attempting  to  infer  whether  or  not  is  was  a  nuclear  submarine;  was 
it  an  SSN  or  an  SS?  The  following  example  illustrates  several  procedures 
used  for  estimating  this  probability. 

Consider  the  two  hypotheses:  the  first  hypothesis,  (H^ ,  is  that 
it  is  an  SSN  and  the  second  hypothesis,  (Hp ,  is  that  it  is  an  SS.  Some 
historical  data  are  relevant  to  this  question.  Six  similar  submarines 
have  been  observed  previously;  five  were  SSNs  and  one  was  an  SS.  The 
task  is  to  add  that  historical  information  to  the  analyst's  theoretical 
knowledge  in  order  to  arrive  at  a  probability  estimate  about  this  particular 
submarine.  This  estimate  was  achieved  through  the  use  of  second-order 
beta  probabilities  and  is  illustrated  in  Figure  1.  The  top  graph  refers 
to  the  percentage  of  SSNs  among  all  submarines  that  might  be  sent  out. 

This  is  represented  by  PfHp  and  is  the  horizontal  axis  of  the  top  graph. 


just  as  likely  that  they  would  send  15%  as  40%  as  80%  SSNs.  It  is  possible, 
through  the  use  of  beta  functions,  to  combine  this  rectangular  prior  probability 
distribution  with  the  historical  frequency  information  in  order  to  obtain  a 
posterior  probability  distribution.  The  two  parameters  of  the  prior  distribution, 
r=l  and  s=l,  are  simply  added  to  the  relative  frequencies  in  order  to  arrive 


at  the  appropriate  parameters  for  the  posterior  probability  distribution:  r=6 

and  s=2.  This  posterior  probability  distribution  is  displayed  in  the  bottom  portion 

of  Figure  1.  It  is  a  probability  distribution  over  the  proportion  of  SSNs. 

The  mean  or  expectation  of  this  probability  distribution  is  .75  and  so  that 
is  the  number  we  selected  as  the  prior  probability  that  this  particular  sub¬ 
marine  was  an  SSN.  That  is  the  number  that  served  as  a  starting  point  for 
analyzing  the  following  data. 

Figure  2  shows  the  logrithmic  chart  that  was  used  in  the  experiment  on 
Bayesian  procedures  described  above.  The  line  at  prior  odds  of  3:1  indicated 
on  the  left  horizontal  axis  and  the  prior  probability  of  .75  shown  on  the 
right  horizontal  axis  indicates  the  starting  point  derived  above. 

The  first  datiu.  is  that  another  submarine  was  sighted  on  a  homeward- 
bound  course.  For  some  well-considered  reasons,  this  datum  slightly 
favors  the  hypothesis  that  the  submarine  being  observed  is  an  SSN;  there  is 
an  estimated  probability  of  .55  that  this  submarine  would  have  been  homeward- 

bound  in  about  this  time  interval  if  the  submarine  being  observed  were  an  SSN;  \ 

1 

there  is  a  50%  probability  estimate  if  this  were  an  SS.  Therefore  the  like-  i 

| 

lihood  ratio  of  the  first  datum,  that  a  homeward  bound  submarine  was  observed,  5 


is  .55/. 50,  which  is  equal  to  1.1.  This  likelihood  ratio  is  now  inserted  into  ] 

i 

the  log  odds  chart  as  shown  in  Figure  3.  It  increases  the  odds  in  favor  of  an  j 


SSN  by  an  almost  imperceptible  amount. 
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fi 
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Figure  3 


Figure  4 


The  second  datum  is  that  the  submarine  being  observed  seems  to  be 
following  a  straight  track  rather  than  taking  evasive  action.  Figure  4 
illustrates  the  manner  in  which  the  likelihood  ratio  associated  with  the 
straight  track  was  elicited.  This  procedure  also  employed  beta  distributions. 
The  upper  portion  of  the  fig  re  refers  to  the  prior  beta  distributions,  ig¬ 
noring  historical  frequencies  of  straight  tracks.  The  left-hand  function  is 
the  second  order  probability  distribution  over  the  proportion  of  straight 
tracks  given  an  SSN  as  estimated  by  the  analyst.  The  analyst  expected  that 
many  more  of  the  SSN,  would  follow  straight  tracks  rather  than  evasive  tracks. 
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The  upper  right-hand  graph  refers  to  the  second  order  probability  distribution  for 
straight  track  given  the  second  hypothesis,  an  SS.  This  is  a  uniform  dis¬ 
tribution.  The  middle  portion  of  the  figure  shows  the  historical  frequencies. 

All  five  SSNs  previously  observed  had  followed  a  straight  track,  whereas  the 
one  SS  did  not.  Addition  of  these  frequencies  to  the  parameters  of  the  beta 
distributions  shown  at  the  top  of  the  figure  imply  the  beta  distributions  shown 
at  the  bottom  of  the  figure.  The  resulting  likelihood  ratio  associated  with 
datum  2  is  therefore  .90,  the  expectation  of  the  bottom  left-hand  distribution, 
divided  by  .33.  This  likelihood  ratio,  2.7,  is  now  added  to  the  log-odds  chart 
as  illustrated  in  Figure  5.  The  observation  that  the  submarine  is  moving  in 
a  straight  track  has  boosted  the  odds  in  favor  of  an  SSN  to  approximately 
9  to  1.  This  second  datum  is  a  very  strong  one,  indeed. 
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Figure  5 


The  analysis  of  the  third  datum  is  displayed  in  Figure  6.  A  political 


event  that  was  expected  to  be  observed  was  not  observed.  A  conditional 
probability  tree  has  been  used  to  estimate  the  likelihood  ratio.  The 
left-hand  branch  of  the  tree  refers  to  the  two  hypotheses,  the  SSN  versus 
the  SS.  The  next  branch  refers  to  the  probability  that  the  expected  event 
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Figure  6. 

would  occur  given  each  of  the  hypotheses.  The  analysts  estimated  that 
the  event  was  certain  to  occur  if  they  were  observing  an  SSN,  but  the 
probability  was  only  .50  that  it  would  occur  given  an  SS.  The  third 
column  of  branches  refers  to  the  probability  of  observing  the  event  if 
it  occurs.  The  analysts  reviewed  some  experimental  literature  and 
concluded  that  there  was  a  60%  chance  that  they  would  observe  the  event 


if  it  actually  occurred,  but  it  was  certain  that  they  would  not  observe 
the  event  if  it  did  not  occur.  That  is,  there  was  no  chance  of  a  false 
alarm.  These  branch  probabilities  implied  the  path  probabilities  displayed 
in  the  boxes  at  the  right-hand  side  cf  the  tree.  Each  probability  was  calculated 
by  taking  the  product  of  the  component  branch  probabilities.  Thus,  the 
.60  in  the  top  branch  is  equal  to  1.0  times  .6.  It  is  now  a  simple  matter 
to  find  the  likelihood  ratio.  The  probability  of  not  observing  the  expected 
event  given  as  SSN  is  equal  to  .40  (the  probability  of  observing  it  if  it 
occurs,  plus  zero,  the  probability  of  observing  it  if  it  doesi't  occur). 

The  probability  of  not  observing  the  expected  event  given  an  SS  is  equal 
to  .70,  so  the  resulting  likelihood  ratio  is  1/1.75. 

This  likelihood  ratio  of  1/1.75  associated  with  not  observing  an 
expected  event  is  now  drawn  on  the  log  odds  chart  in  Figure  7. 
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It  drops  the  odds  in  favor  of  an  SSN  to  approximately  5  to  1. 

The  fourth  datum  is  that  there  has  been  no  contact  with  the 
submarine  for  an  extended  period  of  time.  The  analysts  estimated  a 
probability  of  90%  for  an  extended  lack  of  contact  with  an  SSN,  but 
a  probability  of  only  20%  of  no  contact  with  the  SS.  Accordingly,  the 
likelihood  ratio  associated  with  the  fourth  datum  is  .9/. 2  or  4.5.  This 
likelihood  ratio  is  now  added  to  the  log  odds  chart  as  shown  in  Eigure  8. 
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It  raises  the  odds  in  favor  of  the  SSN  to  about  25  to  1. 

The  fifth  datum  later  turned  out  to  be  a  false  alarm.  An  event 
has  begun  to  develop  and  the  analysts  concluded  that  if  it  continued  to 
develop  it  would  favor  the  hypothesis  that  the  submarine  was'  an  SS.  The 
effect  of  this  datum  is  displayed  on  the  log  odds  chart  in  Figure  9. 
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Several  attempts  yielded  no  appropriate  procedure  for  decomposing  the 
estimate  of  the  likelihood  ratio  associated  with  this  particular  event. 


Figure  9. 

Therefore,  the  analysts  simply  moved  the  odds  on  the  log-odds  chart  on 
an  intuitive  basis.  After  considerable  discussion,  they  decided  that 
if  the  event  did  not  turn  out  to  be  a  false  alarm,  it  favored  the  SS..  it  was 
somewhat  more  diagnostic  than  the  datum  of  not  observing  the  expected 
political  event,  and  not  quite  as  diagnostic  as  the  straight  track.  They 
therefore  moved  the  log-odds  down  just  slightly  less  than  they  had  moved 
it  up  as  a  function  of  the  straight  track.  The  resulting  likelihood 
ratio  turned  out  to  be  2.3.  The  next  morning,  after  completing  this, 
analysis,  the  final  datum  was  identified  as  a  false  alarm  and  the  odds 
estimates  were  therefore  returned  to  about  25  to  1  as  is  shown  in  Figure 
10.  Some  weeks  later,  after  receiving  much  more  information,  it  was 
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Figure  10. 

concluded  that  this  submarine  was  indeed  an  SSN.  It  is  interesting 
to  note  that  four  of  the  analysts  had  discussed  the  problem  thoroughly 
throughout  the  day  and  remained  both  uncertain  and  divided  about  which  of 
the  two  hypotheses  was  correct.  Yet  the  two  analysts  who  made  the  estimates 
for  this  problem  agreed  with  the  .'inal  conclusion,  that  the  data  very  strongly 
favored  the  hypothesis  that  they  were  observing  an  SSN.” 

In  addition  to  providing  a  means  of  combining  expert  judgment  and 
historical  frequencies  this  procedure  serves  as  a  mechanism  for  calibrating 
analysts  with  respect  to  the  assessment  of  likelihood  ratios.  That  is,  if 
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an  analyst  has  a  "feeling"  for  how  diagnostic  some  historical  frequency 
information  is,  and  then  he  observes  a  likelihood  ratio  calculated  by 
means  of  beta  distributions,  be  can  then  use  that  likelihood  ratio  as  an 
anchor  when  assessing  likelihood  ratios  for  which  no  historical  frequencies 
are  available. 

For  many  problems  of  Bayesian  inference,  the  linkage  between  observable 
data  and  the  hypotheses  of  interest  is  indirect.  It  is  then  appropriate  to 
use  hierarchical  or  multi-stage  techniques.  This  requires  the  construction 
of  intermediate  variables  so  that  the  data  can  be  linked  to  intermediate 
variables  which,  in  turn,  can  be  linked  to  the  hypotheses.  We  tested  the 
feasibility  of  this  multi-stage  approach  on  a  quite  complex  problem  of 
submarine  surveillance.  It  provided  a  means  of  handling  conditional 
dependencies  among  data  and  also  for  organizing  the  conclusions  of  the 
analysis.  That  is,  the  hierarchical  procedure  aided  in  assessing  the 
probabilities  of  the  hypotheses  and  also  in  justifying  those  probabilities. 
The  techniques  learned  in  this  feasability  study  are  now  being  incorporated 
in  the  handbook  chapter  on  hierarchical  inference  and  in  a  Special  Issue 
on  hierarchical  inference  that  is  being  prepared  for  Organizational  Behavior 
and  Human  Performance. 

Handbook  for  Decision  Analysis 

A  handbook  for  decision  analysis  is  being  prepared  for  users  of  decision 
theory.  Two  of  the  chapters  ("A  scoring  rule  for  probability  assessment"  and 
"Probability  assessment  for  a  continuum")  are  now  in  printed  form  and  are 
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included  as  enclosures  to  this  final  report.  This  handbook  is  written 
as  a  collection  of  chapters  that  are  relatively  independent  and  can  be  read 
by  the  user  of  decision  theory  for  self  instruction.  The  motivation  for  such 
a  handbook  is  that  users  (both  decision  makers  and  their  staff)  must  under¬ 
stand  decision  analysis  in  order  to  make  efficient  use  of  that  body  of 
knowledge  in  order  to  improve  the  decision-making  process.  The  problem  is 
that  most  texts  on  the  subject  are  overly  mathematical — written  for  the 
decision  theorist  rather  than  the  user. 

Several  chapters  (numbers  6,  8,  S,  11,  and  12  below)  are  being  tested 
in  a  course  on  probability  assessment  for  intelligence  analysts  in  the 
State  Department  that  began  on  26  September  1972.  The  chapters  will  be 
modified  as  a  result  of  use  in  this  and  courses  in  other  governmental 
agencies  designed  for  users  of  probability  assessment  and  decision  analysis. 
The  following  is  a  current  outline  of  the  handbook  chapters. 

Chapter  1  will  provide  a  general  introduction  and  motivation  for  the 
handbook.  It  will  discuss  the  nature  of  the  problems  for  which  decision 
analysis  is  appropriate  and  also  outline  the  book. 

Chapters  2,  3,  and  4  will  present  a  ficticious  case  as  an  illustration 
of  how  decision  analysis  can  be  used  on  a  Navy  problem.  This  case  may  take 
place  during  World  War  II  and  involve  a  decision  about  whether  or  not  a  task 
force  should  be  sent  to  a  particular  destination  when  there  is  a  danger  that 
enemy  submarines  will  be  encountered.  It  will  demonstrate  how  nearly  all 
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of  the  procedures  to  be  described  in  the  later  chapters  can  be  used.  It  is 
thus  intended  to  serve  as  an  introduction  to  those  chapters,  an  introduction 
that  will  permit  the  reader  to  read  through  the  first  four  chapters  of  the 
book  and  then  go  immediately  to  almost  any  of  the  other  chapters  that 
describe  the  problems  or  procedures  with  which  he  may  be  working.  In  order 
to  make  the  reading  both  interesting  and  simple,  the  case  will  be  written  in 
the  form  of  a  dialogue  between  the  Naval  officer  responsible  for  making  the 
decision  and  a  decision  analyst. 

Chapter  5  will  present  an  interpretation  of  personalistic  probabilities 
and  the  justification  for  their  use.  It  will  discuss  the  distinction  between 
relative  frequencies  and  expert  judgment  as  bases  for  probability  forecasts. 

Chapter  6  will  discuss  elementary  rules  of  probability  theory.  It  will 
describe  and  explain  the  three  basic  axioms  and  some  of  the  more  important 
theorems . 

Chapter  7  will  discuss  the  relative  merits  of  different  forms  of  response 
modes  that  can  be  used  when  making  probability  estimates  for  categorical  events. 

Chapter  8  will  discuss  the  assessment  of  continuous  probability  dis¬ 
tributions.  It  will  present  the  relative  merits  of  using  the  fractile  pro¬ 
cedures  (which  assess  areas  under  the  curve)  versus  estimating  density  functions 
(which  focus  on  the  relative  probabilities  of  specific  values).  It  will  also 
discuss  procedures  for  using  discrete  approximations  of  continuous  probability 
distributions. 
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Chapter  9  will  discuss  calibration  with  respect  to  probabilities  estimated 
for  categorical  events.  This  chapter  will  provide  a  scoring  rule  test  which 
the  reader  can  administer  to  himself. 

Chapter  10  will  discuss  calibration  of  continuous  probability  distributions. 
Using  a  definition  that  is  analogous  to  the  discrete  case,  a  continuous  pro¬ 
bability  distribution  is  well-calibrated  if  the  area  of  the  curve  over  any 
specified  interval  reflects  the  percentage  of  the  time  that  that  interval  can 
be  expected  to  occur. 

Chapter  11  will  discuss  conditional  assessment  that  will  focus  primarily 
on  the  use  of  a  probability  diagram  to  decompose  a  probability  assessment. 

Chapter  12  will  discuss  the  revision  of  probabilities  in  the  light  of 
new  evidence.  It  will  describe  Bayes's  theorem,  research  on  the  conservative 
revision  of  probabilities,  and  procedures  for  overcoming  such  conservatism. 

Chapter  13  will  discuss  hierarchical  assessment.  It  will  describe  how 
Bayes’s  theorem  can  be  used  in  a  modified  form  when  multiple  stages  indirectly 
link  the  observable  data  to  the  hypotheses  of  interest.  This  chapter  will  focus 
particularly  upon  principles  to  be  used  when  constructing  a  hierarchical 
inference  tree.  Much  of  it  will  be  based  on  the  on-line  research  on  submarine 
surveillance  described  above. 

Chapter  14  will  introduce  the  concept  of  value  and  show  how  it  is  used 
in  a  decision  analysis. 

Chapter  15  will  discuss  what  is  frequently  called  multi-attribute  utility. 

It  will  describe  procedures  for  reducing  values  that  are  inherently  multi¬ 
dimensional  to  a  single  dimension. 
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Chapter  16  will  discuss  utility  in  its  classical  form  as  a  procedure  for 
incorporating  attitude  toward  risk  into  a  decision  analysis.  It  will  describe 
psychological  procedures  for  eliciting  utility  functions. 

Chapter  17  will  discuss  sensitivity  analysis.  This  is  a  topic  that  is 
repeated  throughout  other  chapters  in  the  book.  It  will  describe  procedures 
that  can  be  used  to  measure  the  degree  to  which  the  conclusion  of  an  analysis 
is  sensitive  to  changes  in  each  of  the  inputs. 

Chapter  18  will  discuss  principles  for  structuring  a  decision  analysis. 

This  chapter  comes  near  the  end  of  the  hand-book  rather  than  at  the  beginning 
because  many  concepts  of  decision  theory  must  be  well  understood  before  a 
sensible  discussion  of  structuring  can  be  understood. 

Chapter  19,  the  last  chapter  of  the  handbook,  will  discuss  practical 
problems  in  the  implementation  and  use  of  decision  analysis  in  Naval  organizations. 


1 

1 

J 

5 

j 

3 

\ 


! 


1 


-22- 


List  of  Technical  Reports  Prepared  Under  This  Contract 

037230-1-T.  The  Application  of  Multi-attribute  Scaling  Procedures  to  the 
Development  of  Indices  of  Value,  1  June  1972,  Michael  F.  O'Connor. 

037230-2-T.  Multi-dimensional  Value  Assessment  for  Decision  Making, 

1  June  1972,  Gregory  W.  Fischer. 

037230-3-T.  Ratio  Versus  Magnitude  Estimates  of  Importance  Factors, 

1  September  1972,  Gregory  W.  Fischer  and  Cameron  R.  Peterson. 

037230-4-T.  A  Scoring  Rule  to  Train  Probability  Assessors,  30  September  1972, 
Jane  Hoffman  and  Cameron  R.  Peterson. 

037230-5-T.  Bayes's  Theorem:  Response  Scales  and  Feedback,  30  September  1972, 
Patricia  A.  Domas,  Barbara  C.  Goodman  tid  Cameron  R.  Peterson. 

037230-6-T.  Four  Methods  for  Assessing  Multi-attribute  Utilities:  An 
Experimental  Valication,  30  September  1972,  Gregory  W.  Fischer. 
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